Why is s3 handled specially here? Do we need to do anything for other filesystems?

One way to make this clearer could be:

if not storage_options:
    logger.info(f"Using default storage options for `{protocol}` filesystem.")
    if protocol == S3:
        storage_options = S3RemoteStorageOptions().get_storage_options()

 try:
    return fsspec.filesystem(protocol, **storage_options)
...

Following up on my comment below, I now believe that this storage_options handling shouldn't be necessary.

Also, this approach has an additional bug, in that it will break existing use_credentials behavior by overriding it with credentials from the environment (when both are using s3).

@tgaddair S3 is handled especially here because users may want to pass in a different S3 compatible storage like Minio. The issue is that to do this, users need to pass in the --endpoint_url flag to AWSCLI because AWSCLI currently doesn't have an environment variable that can be configured to pick up this URL automatically (although they are currently working on a proposal to add it in: aws/aws-sdk#229). Within fsspec, that would involve manually passing in client_kwargs with the endpoint_url defined in the object. I think this is the only reason to do something special for S3 at the moment.

I can't entirely say what would be required for other file systems yet, but those should automatically get created via fsspec.

About the use_credentials method - I think that this actually won't be an issue because both use_credentials and this approach will end up using the same key and secret since they're derived from the same environment variables. The only thing that changes is the endpoint URL, which in either case would need to change for S3 compatible storage since use_credentials would also require this new endpoint URL (if it is specified).

Doesn't make sense to me to have mlflow coupling here. Looks like the idea here is to pull storage options from certain environment variables when they're not passed in explicitly. However, this shouldn't be necessary, as fsspec already has a mechanism for plumbing credentials through the environment, either through the use_credentials helper that sets the conf dictionary, or through the FSSPEC_ env vars:

https://filesystem-spec.readthedocs.io/en/latest/features.html#configuration

So I would rely on the built-in fsspec plumbing mechanisms instead of recreating it here.

@tgaddair the MLFLOW_S3_ENDPOINT_URL is actually just a placeholder name and can be change to any other name. It doesn't actually end up involving any coupling with MLFLOW. I'll rename this environment variable to something more generic like S3_ENDPOINT_URL to make this more clear!

I like the idea of using the FSSPEC_ env vars as a potential substitute to avoid plumbing S3 credentials in this particular case with endpoint URLs. Will update and clear this up so that we don't need to do any of this manually.

Although following from the thread here (fsspec/s3fs#432) where you've requested this before as well, it doesn't seem like there's a neat way to specify the endpoint URL since it needs to be set inside a nested dictionary. How do you feel about just using this as an environment variable for now but with a different name?

• Merge into _get_remote_checkpoint_dir with optional argument "tmp" to avoid code duplication